What is claimed is: 



1. A method for tracking motion of a face comprising the steps of: 
selecting salient features of the face for motion tracking; and 
tracking motion ofWie salient features of the face. 
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2. The method of claim IV further comprising: 

acquiring a plurality of mitial 2-D images of the face; 
calculating 3-D locationsW the salient features; and 
determining a surface normal for each salient features. 
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3. The method of claim 1 further Comprising: 

receiving a chronologically ordered sequgiice of 2-D images of the face in action; and 
locking onto the salient features. 



The method of claim 1 further conjprrsmg 
tracking 3-D global motion of thoHTace in eac 



tracking 3-D local motion of the/face in each\image. 



h image; and 



20 5. The method of claim 1 wherein the step of ^electing comprises fixing markers to the 
face and the step of tracking comprises tracking the motion of the markers. 
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6. The method of claim 5 wherein a first set of matkers identifies global motion and a 
second set of markers identifies local motion of theVace. 

7. The method of claim 4 wherein the step of tracking tn^ 3-D global motion comprises 
the steps of: 

predicting the location of global salient features in a 2-D i^iage; 
detecting global salient features in the 2-D image; and 
estimating the 3-D global motion of the face in the 2-D image 



8. The method of claim 7 wherein the step of estimating comprises Calculating the 
position and shape of the face to conform to the 3-D locations and uie detected 
locations of the global markers under a perspective projection mode* 
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9. The method of ci^aim 4 wherein the step of tracking the 3-D local motion comprises 
the steps of: 

predicting the locati<3n of local salient features; 
5 detecting local salient features; and 

estimating the 3-D local motion of the face. 

10. The method of claim 9 wherain the step of estimating comprises: 
finding 3-D locations of lo^Snjprkers to conform to the detected 2-D locations of the 

10 local markers; and 

calculating an action vector representing the weights of facial actions in the 2-D 
image conforming to the found 3-D locations of local markers and the 3-D locations 
of the local markers for the neutral anaSJhe action states under a perspective projection 
model. 



1 1 . A method for tracking motion of a face comprising the steps of: 
determining the calibration parameter of a camera; 
selecting salient features on the face for motion tracking; 

C3 acquiring a plurality of initial 2-D images of the face; 

qj 20 calculating 3-D locations \f the salient features in accordance with the calibration 

parameter of the camera; \ 

determining a surface normalNfor each salient features; 

receiving a chronologically orcrered sequence of 2-D images of the face in action; 
tracking motion of the face in eafch 2-D image; and 
25 storing or transmitting tracked motion of the face. 

12. The method of claim 1 1 further comprising the steps of: 
locking onto the salient features; and 

detecting loss of lock and hence the need for re-locking onto the salient features. 

30 

^ X 13. The method of claim 1 1 wherein the step of tracking comprises the steps of: 
A^> / tracking the 3-D global motion pf the face in each image; and 

tracking the 3-D local motion of the face in each image. 



7 14. The method of claim 11 comprising the further step of repeating the locking and 
tracking steps after the detecting step. 

15. The method of claim 11 wherein the step of selecting comprises recognizing salient 
5 facial features and the step of tracking comprises tracking the motion of the salient 

facial features. 

16. The method of claim 1 1 wherein the step of selecting comprises fixing markers to the 
face and the step of tracking comprises tracking the motion of the markers. 

10 

17. The method of claim 16 wherein a first set of markers identifies global motion and a 
second set of markers identifies local motion of the face. 

18. The method of claim 17 wherein the markers comprise at least two colors. 
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19. The method of claim 18 wherein the two colors are contrasting. 

20. The method of claim 19 wherein the colors are black and white. 

20 21. The method of claim 16 wherein the markers comprise two concentric circles of 
different colors. 

22. The method of claim 21 wherein the outer circle has a diameter at least twice the 
diameter of the inner circle. 
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23. The method of claim 11 wherein the step of selecting comprises wearing a head-set 
with markers. 

24. The method of claim 23 wherein the head-set comprises a strap for a chin. 

30 

25. The method of claim 23 wherein the head-set comprises a strap for eyebrows. 

26. The method of claim 23 wherein the head-set comprises at least one strap for a skull. 
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27. The method of claim 1 1 wherein the acquired 2-D images include at least two views 
of the face with markers in a neutral state at different orientations; 

28. The method of claim 27 wherein the two views are orthogonal. 

5 

29. The method of claim 1 1 wherein the acquired 2-D images comprise front, forehead, 
chin, angled-right, angled-right-tilted-up, angled-right-tilted-down, angled-left, 
angled-left-tilted-up, angled-left-tilted-down, fiill-right-profile, full-right-profile- 
tilted-up, full-right-profile-tilted-down, full-left-profile, full-left-profile-tilted-up, and 

10 full-left-pro file-tilted-down views of the face with markers in the neutral state. 

30. The method of claim 1 1 wherein the acquired 2-D images comprise front, forehead, 
chin, full-right-profile, and full-left-profile views of the face with markers in the 
neutral state. 
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^^U] \ 31. The method of claim 1 whereinVhe acquired 2-D images include a plurality of views 
fUJ: m/ of the face with markers in at least one action state. 

□ 32. The method of claim 31 wherein the action states of the face comprise smiling lips, 

n | 20 kissing lips, yawning lips, raised eyebrows, and squeezed eyebrows. 

33. The method of claim 31 wherein the acquired 2-D images of the face in an action 
state include at least two views at different orientations. 

25 34. The method of claim 33 wherein the two views are front and angled-right. 

35. The method of claim 1 1 wherein the step of selecting comprises fixing markers to the 
face and the step of calculating comprises calculating the 3-D locations of the markers 
placed on the face. 
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36. The method of claim 35 wherein the step of calculating the 3-D locations of the 
markers comprises the steps of: 

calculating the 3-D locations of the global and local markers in the neutral state; and 
calculating the 3-D locations of the local markers in each action state; 
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37. The method of claim 36 wherein the step of calculating the 3-D locations of the 
global and local markers in the neutral state comprises the steps of: 
calculating the 3-D locations of the markers to conform to their 2-D locations in the 
5 2-D images of the face in the neutral state under an orthographic projection model; 

calculating relative distances of the face to the camera in the 2-D images to conform 
to the 2-D locations of the markers and their calculated 3-D locations under a 
perspective projection model; 

modifying the 2-D locations of the markers to conform to the calculated relative 
10 distances and the 3-D locations under a perspective projection model; 

recalculating the 3-D locations of the markers to conform to their modified 2-D 

locations under an orthographic projection model; 
O repeating the steps of calculating the relative distances, modifying the 2-D locations, 

rff and recalculating the 3-D locations to satisfy a convergence requirement; and 

^ 15 translating and rotating the 3-D locations so that they correspond to a frontal-looking 

U1 face. 

: a 38. The method of claim 36 wherein the step of calculating the 3-D locations of the local 

O markers in each action state comprises the steps of: 

20 estimating the orientation and position of the face in each 2-D image of the action 

y state to conform to the 3-D and 2-D locations of the global markers under a 

perspective projection model; and 

calculating the 3-D locations of the local markers to conform to the estimated 
orientation and position of the face and the 2-D locations of the local markers under a 
25 perspective projection model; 

39. The method of claim 13 wherein the step of tracking the 3-D global motion comprises 
the steps of: 

predicting the location of global salient features in a 2-D image; 
30 detecting global salient features in the 2-D image; and 

estimating the 3-D global motion of the face in the 2-D image. 

40. The method of claim 39 wherein th§ step of predicting comprises calculating 2-D 
locations of the global salient features under a perspective projection model using the 
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position and orientation of the face in a previous 2-D image, and the step of detecting 
comprise detecting the global markers. 

41. The method of claim 40 wherein detecting the global markers comprises: 
determining visibility indices of global markers; 

designing correlation filters for the global markers; 

detecting the global markers by applying elliptical correlation filters in a 

neighborhood of the global markers; and 

eliminating superfluous and multiple detected locations. 

42. The method of claim 39 wherein the step of estimating comprises calculating the 
position and shape of the face to conform to the 3-D locations and the detected 
locations of the global markers under a perspective projection model. 

43. The method of claim 13 wherein the step of tracking the 3-D local motion comprises 
the steps of: 

predicting the location of local salient features; 
detecting local salient features; and 
estimating the 3-D local motion of the face. 

44. The method of claim 43 wherein the local markers are placed on eyebrows and lips. 

45. The method of claim 44 wherein the locations of the local markers comprise 
proximate ends of the eyebrows, comers of the lips, and the upper and lower centers 
of each lip. 

46. The method of claim 43 wherem the step of predicting the locations of local markers 
comprises calculating the locations of the local markers using the position, 
orientation, and action values ofl the face in a previous 2-D image and the step of 
detecting comprise detecting the global markers. 

47. The method of claim 44 wherein defecting the local markers comprise: 
determining visibility indices of local markers; 

designing correlation filters for the local markers; 
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detecting the local markers by applying elliptical correlation filters in a neighborhood 
of the local markers; and \ 

eliminating superfluous and multiple detected locations. 

48. The method of claim 43 whirein the step of estimating comprises: 

finding 3-D locations of local markers to conform to the detected 2-D locations of the 
local markers; \ 

calculating an action vector representing the weights of facial actions in the 2-D 
image conforming to the found b-D locations of local markers and the 3-D locations 
of the local markers for the neutral and the action states under a perspective projection 
model. \ 

49. The method of claim 48 wherein the step of calculating an action vector comprises the 
steps of : 

calculating the difference between the 2-D locations of the local markers detected in 
an image and the 2-D locations of the same markers corresponding to the neutral face; 
modifying the difference to conform to the orthographic projection; 
calculating the 3-D displacements of the local markers with respect to their location in 
the neutral face; and 

calculating the amount of facial actions conforming to the 3-D displacements of the 
local markers. 

50. The method of claim 48 wherein the step of calculating an action vector comprises the 
steps of : 

calculating the 2-D locations of the local markers corresponding to the neutral face 
using the global motion found for the current image; 

calculating the 2-D locations of the local markers corresponding to the action faces 
using the global motion found for the current image; 

calculating the distance between the detected locations, the distance between the 
neutral locations, and the distance between the action locations of the markers at the 
right and left comers of the lips; 

calculating the distance between the detected locations, the distance between the 
neutral locations, and the distance between the action locations of the markers at the 
upper and lower center of lips; 



39 



calculating the distance between the detected locations, the distance between the 
neutral locations, and the distance between the action locations of the markers at the 
proximate ends of the eyebrows; 

determining the fractional displacements of the local markers for the lips area and for 
the eyebrows area; and 

determining action mode and amount for the lips area and for the eyebrows area based 
on the fractional displacements of the local markers. 
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